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, Abstract. The well known formula of B. Fuglede expresses the 

mean value of the Radon k-plane transform on M." as a Riesz po- 
£f) [ tential. We extend this formula to the space ofnxm real matrices 

and show that the corresponding matrix k-plane transform / — > / 
is injective if and only if n — k > m. Different inversion formulas 
• for this transform are obtained. We assume that / £ LP or / is 

ry ^ a continuous function satisfying certain "minimal" conditions at 

infinity. 



> 

(N 



1. Introduction 

In 1958, B. Fuglede [Fu] proved the following remarkable formula 

(1.1) c(fnx) = (I k f)(x), x G IR n , 

where / = /(r) is the integral of f(x) over a fc-dimensional plane r, < 
k < n; (/) v (x) is the mean value of /(r) over all fc-planes through x, 
^ [ (I k f)(x) denotes the Riesz potential of order k, and c = c(n,k) is a 

constant. For k — n — 1, when r is a hyperplane, this formula was 
implicitly exhibited by J. Radon [R] who was indebted to W. Blaschke 
for this idea. A consequence of (jl.lj) is the inversion formula 
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1.2) / = c(-A) k /\f) 



> 

in which A denotes the Laplace operator. 

Our aim is to extend (jl.lj) and (jl.2j) to the case when x is an n x m 
real matrix. We note that Riesz potentials of functions of matrix argu- 
ment and their generalizations arise in different contexts in harmonic 
analysis, integral geometry, and PDE; see jFar j . [FK], [Ge], [Kh], [Ra], 
[Stl], [Shl]-[Sh3]. A systematic study of Radon transforms on ma- 
trix spaces was initiated by E.E. Petrov [Pel] and continued in [C], 
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[Gr], [Pe2]-[Pe4], [Shi], [Sh2]. These publications traditionally deal 
with Radon transforms / — > f of C°° rapidly decreasing functions, and 
employ decomposition in plane waves in order to recover / from /. 

We suggest another approach which is based on a matrix analogue 
of (jl.l)) (see Theorem 5.4) and allows to handle arbitrary continuous 
or locally integrable functions / subject to mild restrictions at infin- 
ity. These restrictions are minimal in a certain sense. We show that 
inequality n — k > m is necessary and sufficient for injectivity of the 
matrix fc-plane transform, and obtain two inversion formulas (in terms 
of the Fourier transform and in the form (|1.2jl ). 

Our key motivation is the following. Converting integral-geometrical 
entities into their matrix counterparts has specific "higher rank" fea- 
tures and sheds new light on classical problems in multidimensional 
spaces (just note the space M> N with N = nm can be treated as a 
collection ofnxm matrices). 

Acknowledgements. We are grateful to Prof. E.E. Petrov and Dr. 
S.P. Khekalo for the papers they sent us, and useful discussions. 

2. Preliminaries 

We establish some notation and recall basic facts. Let SH„ im be 
the space of real matrices x = (xij) having n rows and m columns. 
We identify 9Jt„ im with the real Euclidean space M. nm and set dx = 
nr=i llj=i dxij. m the following x' denotes the transpose of x, I m is 
the identity m x m matrix, stands for zero entries. Given square 
matrices a,b,q,s, etc., we denote by \a\, \b\, \q\, \s\ their determinants. 
GL(m, R) is the group of real non-singular m x m matrices; SO(n) is 
the group of orthogonal n x n real matrices of determinant one, en- 
dowed with the normalized invariant measure. We denote by V m the 
cone of positive definite symmetric matrices r = (r^-); tr(r) is the trace 
of r, dr = Yli<jdrij. The Lebesgue space L p (jSl ntTn ) and the Schwartz 
space S(Wl nim ) are identified with respective spaces on M nm . 

Lemma 2.1. (see, e.g., [Mu, pp. 57-59]). 

(i) If x = ayb, where y £ 9ft n>m , a £ GL(n, R), b £ GL(m,M), then 
dx = \a\ m \b\ n dy. 

(ii) If r = q'sq, where s £ V m , q £ GL(m, R), then dr = \q\ m+1 ds. 
(Hi) If r = s^ 1 , where s £ V m , then r £ V m , and dr = |s| _m-1 <is. 

The Siegel gamma function associated to the cone V m is defined by 
(2.1) r m (a) = / exp(-tr(r))|r| a " d dr, d=(m+l)/2. 
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This integral converges absolutely if and only if Re a > d — 1 , and 
represents a product of ordinary T-functions: 

(2.2) r ra (a) = 7r-^- 1 )/ 4 r( a )r(a - \) . . . r(a - ^). 

If 1 < k < m, fcGN, then 

(2.3) r m (a) = 7r fe ( m - fc )/ 2 r fc (a)r m _ fc (a - fc/2), 

f2 4 x r m (a) = r fc (a + (fc-m)/2) 

1 ' J r m (a + fc/2) T fc (a + fc/2) 

For n > m, let \4, m = {«£ $Hri, m : v'v = I m } be the Stiefel manifold 
of orthonormal m-frames in R n ; V n ^ n = 0(n) is the orthogonal group 
in W 1 . We fix invariant measure dv on V n>m [Mu, p. 70] normalized by 

/2m^nm/2 
dt 



v„ 



T m (n/2) 



Lemma 2.2. (polar decomposition; see, e.g., [Mu, pp. 66, 591], [Ma]). 
Let x G 9R n ,m, n> m. //rank(x) = m, then 

x = vr 1/2 , v G Ki,m, r = x'x G P m , 
and dx = 2- m |r|( n - m - 1 )/ 2 rfrrft;. 

3. RlESZ POTENTIALS 

For x G 9Jtn, m , let |x| m = det(x'x) 1 / 2 . The Riesz potential of order 
a G C is defined by 

(3.1) (/«/)(x) = [ f(x-y)\y\ a m - n dy, 

_ 2 am 7r" m / 2 r m (a/2) 
T m ((n - a)/2) 

provided this expression is finite. Note that for m > 2, the sets of 
poles of T m (a/2) and T m ((n — a)/2) are {m — 1, m — 2, . . .}, and 
{n — m+1, n — m + 2,...} respectively. These sets overlap if and only 
if 2m > n + 2 (keep this inequality in mind!). For m — 1, the set of 
poles of T m ((n — ol)/2) is {n, n + 2, n + 4, . . .}. Thus if 

n — m + 1, n — m + 2, . . . for m > 2, 
n, n + 2, . . . for m = 1, 

then the coefficient l/7„, m (a) is infinite. In the following we exclude 
these values of a and focus on the case m > 2. 



a 
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The Riesz distribution corresponding to (J3.1j) is denned by 



(3.2) (h a , f) = a.c. j h a (x)f(x)dx, h a (x) 



\x 



a—n 
m 



7n,m(«) ; 



where "a.c." abbreviates analytic continuation in the a- variable, and 
/ G S(%R nim ). The following lemma resumes basic properties of h a . 

Lemma 3.1. Let m > 2. 

(i) The integral in absolutely converges if and only ifRea > m—1. 

(ii) The function a —>■ h a extends to all a G C as a meromorphic S'- 
distribution with the only poles at the points a = n—m+1, n—m+2, 
The order of these poles is the same as in T m ((n — a)/ 2). 

(hi) h a is a positive measure if and only if a belongs to the set 

(3.3) W = {0,l,2,...,k }U{a:Rea>m-l] a^n-m(mod 1)}, 

ko = min(m — 1, n — m). 
(iv) // a 7^ n — m + 1, n — m + 2, . . and 

(3.4) (Ff)(y)= J exp(tr(iy'x))f(x)dx 

is the Fourier transform of f G S(M niTn ), then 
(3-5) (haJ) = (2n)- nm (\ y \- a ,(Ff)(y)) 

in the sense of analytic continuation. In particular, 

(3.6) (to,/) = /(0). 

These statements can be found in [Kh]; see also [Shi], [Sh3], [FK]. 
The set ()3.3|) is an analog of the Wallach set in [FK], p. 137. We adopt 
this name for our case too. The formula (J3.5|) is known as a functional 
equation for the zeta function a — > (h a ,f). Different proofs of (|3.5|) 
(usually in a more general set-up) can be found in [Ge], Chapter IV; 
[Ra], Prop. II-9; [Shi], [Far], [FK], Theorem XVI.4.3; see also [Stl]. 

In the case 2m < n + 2, when poles of T m (a/2) and T m ((n — a)/2) 
do not overlap, a simple proof of (|3.5J) can be given following slight 
modification of the argument from [St2], Chapter III, Sec. 3.4. Let 
e s {x) = exp(tr(— xsx')/4:7i), s G V m . By the Plancherel formula, 

(3.7) | S |""/ 2 J (Tf)(y)ex V (tr(-7Tys- l y'))dy= J f(x)e s (x)dx. 
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Multiplying ()3.7|) by |s|( n a ^ 2 d , d = (m + l)/2, and integrating in 
s G V m , after changing the order of integration we obtain 



(J r f)(y)a(y)dy = / f(x)b(x)dx 



where 



-a/2- 



! exp(tr(— irys 1 y'))ds (s = t v ) 



j \t\ a / 2 - d exp(tr(-nty'y))dt = T m (a/2) 



-am/2 1 I -a 
I y I m 



if i?e a > m — 1 , and 



b(x) = I |.s 
7>, 



(n-o)/2-d 



e s (x)ds 



^m(n-a)/2 T m ((n - «)/2) 



;r 



if i?ea < n — m + 1. A simple computation of these integrals is 
performed using ()2.1|) and Lemma 12.11 Thus (|3.5|) follows if the set 
m — 1 < i?e a<?2 — m + lis not vacuous, i.e., 2m < n + 2. We note 
that if 2m > n + 2 then distributions in both sides of ()3.5|) are not 
regular simultaneously. 

Explicit representations for h a in the discrete part of the Wallah set 
(13. 3|) play a vital role in our consideration. 

Lemma 3.2. Let f G «S(9Jtn >m ), fco = min(m— 1, n — m), m > 2. JTien 
for k = 1,2,..., k , 



(3.1 



(h k , f) = c J du j f ( 7 

2Tt fc , m SO(n) 







d7, 



(3.9) 

Proof. We split x G 97t n)m in two blocks x = [y; 6] where y G 9Jl n) fc and 
b G 97t njm _fc. Then for Re a > m — 1, 

(a-n)/2 



ln,m{.OL) 



dy 



y'y y'b 
b'y b'b 



db 



where 



denotes the determinant of the respective matrix 



By passing to polar coordinates (see Lemma l2~2|) y = t>r 1//2 , v G V n ^ 
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r e Pfc, we have 

(KJ) 



7n,m(a) 



J dv J \r\ in - k ^/ 2 dr 



v n , k v k 
f([vr 1/2 ;b\) 



-k 



r r l l 2 v'h 
y vr i/2 y h 



(a-n)/2 



db 



n,k 



7n,m(a) 



dl / | r |(n-fc-l)/2 dr 



SO(n) V k 



A([V 1/2 ;&]) 



Here 



We write 



A = 



l l 2 \'b 



b'\ r l l 2 b'b 



(a-n)/2 



db. 



h 




61 

Since \' b = bi, then 



(ha J) 



X 



Note that 



2 k O r n,k 
7n,m(a) 



d 1 J \r\( n ~ k ~ l V 2 dr J dh 
r 



1 



a/2 bi 
b 2 



r l/2 6l 



& ; r i/2 b'^ + b'^2 



(a-n)/2 



<i& 2 



n — k,m — k 



r i/2 6i 



fe^r 1 / 2 6i6i + ^&2 



r 

u l' 1 m—k 



h r-^b, 
6' 2 6 2 



and 



see, e.g., [Mu], p. 577. Therefore, 



det(r)det(6 2 6 2 ), 



(3.10) (h a ,f)=c a J d^ J \r\^- k - l ^ 2 dr J ^ a _ k ( 1: rM)db u 



SO(n) V k 



k,m — k 



where 
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2 k a ntk J n -k,m-k(at - k) 



ln,m{a) 



ln-k,m-k{a - k) 

m 

(oc — k) — (n — k) 

x \b 2 b 2 \ 2 db 2 . 



r l/2 ^ 

6 2 



n — k.m — k 



The last expression is the Riesz distribution of order a — k in the b 2 - 
variable. Owing to ()3.6|) . analytic continuation of f)3.10j) at a — k 

reads 



(h a ,f) = c k / d 7 / |r|- 1/2 rfr / /, 



SO(n) V k 



art* 



r l/2 &1 

o o 



Cfc = lim c a . To transform this expression, we replace 7 by 7 



a—fk 



f3 G SO{k) and integrate in (3. This gives 



(3 

in-fc 



= c k \r\- l ' 2 dr / 



/Jr 1 / 2 (3b x 




(set C = 77 = 6|r| 1//2 and use Lemma f2.2jl 



&k,k 



dv / d( f. 



M k . k Tt, m _ k SO(n) 



v C 




G?7 



<■ I du J / ( 7 

2« fc , m SO(n) 



It 





d7, 



°k,k a ^ k 7n,m(a) V 2 / V2 



(here we used formulae ()2.3|) and (|2.4jl ). 
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Definition 3.3. According to Lemma 13.21 and (jH.fijl . we can redefine 
the Riesz potential I a f for any locally integrable function / as 



(3.11) 



(I a f)(x) 



f(x-y)\y\^ n dy 



if Re a > m — 1; a n — m+l,n — m + 2, 

c?7 if a = 1, . . . , k , 



<■ i du J / ( x — 7 







Here m > 2, c is the constant (|3.9|h fco = min(m — l,n — m). It is 
assumed that / is good enough, so that the corresponding integrals are 
absolutely convergent. 

Conjecture 3.4. We state the following hypotheses. 

(a) For / e L p (97t n m ), the integrals in ()3.1H) absolutely converge if and 

only if 



(3.12) 



l<p<(n + m — 1)/ (Re a + m — 1) . 



(b) If / is a continuous function satisfying f(x) = 0(\I m + x'x\^ x ^ 2 ), 
then the absolute convergence of ()3.11|) holds if and only if A > Re a + 
m — 1. 

This conjecture is true if a is a positive integer; see the next section. 
Remark 3.5. Another formula for hf. obtained in [Shi] and [Kh] reads 



(3.13) (h k J)= Cl 



dy 



\y\k 



f{[y,yz])dz, k = l,2,...,k . 



(3.14) 



Cl 



fn — m\ , ( k 



It can be derived from ()3.8|) . Indeed, 
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(h k ,f) 



< I d u j f ( 7 

M Km SO(n) 



u 




G?7 



c j du J f(>y\ u)d<y ( A 

m Km SO(n) 
C 



h 





G K 



71, fc 



du / f(vu)dv. 



Now we represent w in the block form it = [77; £], 77 G SPlfc^, £ G 9ttfc jm _fc, 
and change the variable £ = r/z. This gives 



(h k , f) 



ri\ m - k dr] 



dz \ f(v[rj;7]z])dv. 



Using Lemma \2. 21 repeatedly, and changing variables, we obtain 



(h k ,f) 



Cl 



Vk 

dy 



r |(m-fc-i)/2 dr j dz j f(v[r^ 2 y^z])dv 



is/1* 

where by (pT5Jl . (jZHj) and (jOjl . 

ci = ^ = 2 _ fcV ( t - n . m)/2r Jn-m| /r/ 



4. Radon transforms 



4.1. Matrix planes. Let k,n, and m be positive integers, < k < n, 
V n ,n- k be the Stiefel manifold of orthonormal in — fc)-frames in R n . For 
£ G V^ n.fc, t G 2t n -fc,m, the linear manifold 



(4.1) 



r = t(£, t) = {xe OT n , m : gx = t} 



will be called a matrix k-plane in 3Jt„ )rn . We denote by G(n, k, m) a va- 
riety of all such planes. The parameterization r = r(£, i) by the points 
(£, i) of the "matrix cylinder" V nt n- k x 9ttn-jfc,m is not one-to-one because 
for any orthogonal transformation G 0(n — fc), the pairs (£, t) and 
(£9', 9t) define the same plane. We identify functions <p{r) on G(n, k, m) 
with functions (p(£, t) on V n>n - k x 9Jl n _fc,m satisfying <p(£9', 9t) = ip(£, t) 
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for all 6 G 0(n — k), and supply G(n,k,m) with the measure dr so 
that 



(4.2) 



<P{t) dr = 

G(n,k,m) V„ 



(p(£,t) d£dt. 



-feX 



Wl 



n — k,m 



The matrix /c-plane is, in fact, a usual /cm-dimensional plane in the 



Euclidean space M. nm . To see this, we write x 
t = (t it j) G Wl n _k,m as column vectors 

Xl,2 



X 



i -J i 



(4.3) 



x 



\ 



t 



J 



*1,2 



y tn—k,m J 



>(n— fe)m 



and 



and denote 

(4.4) £ = diag(£, Km,(n-fc)m. 

Then (fO reads 

(4.5) r = r(f , t) = {x € R nm : l'x = t}. 

The km- dimensional planes ()4.5|) form a subset of measure zero in the 
affine Grassmann manifold of all km- dimensional planes in M nm . 

The manifold G(n, k, m) can be regarded as a fibre bundle the base 
of which is the ordinary Grassmann manifold G n ^ of fc-dimensional 
linear subspaces rj of M n , and the fibres are homeomorphic to 9Jt n _fc, m . 
Indeed, let ir : G(n, k, m) — > G n ^ be the canonical projection which 
assigns to each matrix /c-plane t(£,£) the subspace 

(4.6) V = r?(0 = {yeU n : £'y = 0} G G n , fc . 

Let r; be the orthogonal complement of?? in IR n . The fiber H v = 7T _1 (?7) 
is the set of all matrix planes (j4.1j) . when i sweeps the space Wl n -k, m . 

Regarding G(n, k, m) as a fibre bundle, one can utilize a parameter- 
ization which is alternative to (j4.1|) and one-to-one. Let 

(4.7) x = [x 1 ...x m ], Xi G M n , t=[ti...t m ], t,GM n - fc . 
For r = r(£, t) G fc, m), we have 

r = {x G 9Jt„, m : i'xi = U, i = 1 . . . m}. 

Each k- dimensional plane r< = {xj G M n : = tj} can be parame- 
terized by the pair (77, Xi), where 77 is the subspace (|4.6|) . and Aj G t? -1 , 
z = 1, . . . , m, are columns of the matrix A = £t G 9Jt n ,m- The corre- 
sponding parameterization 



t] g G U;k , A = [Ai . . . A r 



Ai G rj ± , 



THE FUGLEDE FORMULA 



11 



is one-to-one. 



4.2. Definition of the Radon transform. The matrix fc-plane Radon 
transform / of a function f(x) on Wl n . m assigns to / a collection of in- 
tegrals of / over all matrix planes r G G(n, k, m). Namely, 



/(t) 



/(*)■ 



In order to give this integral precise meaning, we note that the matrix 
plane r = r(£, t), £ G V n , n -k, t G 9# n -fc,m, consists of "points" 



x 



9i 



u 
t 



where u G SHfc,m, and g% G SO{n) is a rotation satisfying 



(4.9) g& =£, £ = 

This observation leads to the following 





In—k 



n,n—k- 



Definition 4.1. The Radon transform of a function f(x) on Wl n , m is 
defined as a function on the "matrix cylinder" Ki,n-fc x ffl n _k,m by the 
formula 



(4.10) 



f(r) = m,t) 



fix 



u 
t 



du. 



The reader is encouraged to check that ()4.10|) is independent of the 
choice of the rotation g% : £o - ► £• I n terms of the one-to-one parame- 
terization (|4.8p . where r = r(^, A), 77 G (j n ,fc, A = [Ai . . . A m ] G 9tt n ,m, 
and Aj G r/ -1 , the Radon transform (|4.1(J|) reads 



(4.11) 



/(r) = / dy x . . . / /([yi + Ai . . . y m + X m ]) dy m . 



If m = 1, then /(£,£) is the ordinary /c-plane Radon transform that 
assigns to a function /(x) on R™ a collection of integrals of / over all 
fc-dimensional planes [Hel]. A different definition of the matrix Radon 
transform was given by E.E. Petrov [Pel]-[Pe3] (the case n — k = m), 
and L.P. Shibasov [Shi], [Sh2] (the general case). 
The following properties can be easily checked. 

Lemma 4.2. Suppose that the Radon transform 

f(x) ► /(£,*), X G 9Jl n ,m, (£, t) G V n , n -k X ^n-k,m, 
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exists (at least almost everywhere). Then 

(4.12) me',et) = fat), w g o(n - k). 

If g{ x ) — 1 X P + V where 7 G 0(n), [3 G 0(m), ?/ G 9K n ,m; t/ien 

(4.13) (/°s) A (£,*) = /(7^ + £Vl/)- 
In particular, if f y (x) = f(x + y), then 

(4.14) f y (U) = m,t'y + t). 

The equality ()4.12|) is a matrix analog of the "evenness property" of 
the classical Radon transform, cf. [Hel], p. 3. 

Lemma 4.3. 

(i) If f G L 1 (2H ni?n ) ; then the Radon transform /(£,£) exists for all 
£ G V^.n-fc and almost all t G 97t n -fc,m. Furthermore, 

(4.15) y f{£,t)dt = J f(x)dx, V£GV n , n _ fc . 

(ii) Let ||x|| = (tr(x'x)) 1 / 2 = (xf 1 + . . . +^ iJn ) 1//2 . /// is« continuous 
function satisfying 

(4.16) /(x) = 0(||x|r a ), a>A;m, 
i/ien /(£,£) exists /or a// £ G V^ )Tl _^ and a// 1 G Tl n ~k,m- 
Proof, (i) is a consequence of the Fubini theorem: 



m,t)dt 



dt 
Mi 

f(g^x) dx 



f[9t 



t 



du 



f(x) dx. 



(ii) becomes obvious if we regard r = r(£, t) as a /cm-dimensional plane 
(1431) in M nm . □ 

A much deeper result is contained in the following 

Theorem 4.4. If f G L p (OT„ im ) t/ien t/ie Radon transform f{£,,t) is 
finite for almost all (£, i) G Kj in _fe x 97t n _ fc m provided 

n + m — 1 



(4.17) 



1 < p < p 



A; + m — 1 

If f is a continuous function satisfying f(x) = 0(\I m + x'x\~ x / 2 ), A > 
k + m-1, then /(£, £) zs /mrfe /or a// (f , t) G V^n-fc x ^n-fe.m- 
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The proof of this theorem was given in [OR] using Abel type repre- 
sentation of the Radon transform of radial functions. The conditions 
for p and A are sharp. For instance, one can show [OR] that 

(4.18) f (x) = \2I m + x'x|- (n+m - 1)/2p (log \2I m + x'xiy 1 

belongs to L p (ffl n ^ m ), and /o(£, t) = oo if p > po. For m — 1, the result 
of Theorem 14.41 is due to Solmon [So]; see also [Ru2] for another proof. 



4.3. Connection with the Fourier transform. The Fourier trans- 
form of a function / G L 1 (M„ >m ) is defined by (|3.4j) . The following 
statement is a matrix generalization of the so-called Central Slice The- 
orem. It links together the Fourier transform (J3.4)) and the Radon 
transform (j4.1Uj) . In the case m = 1, this theorem can be found in [Na, 
p. 11] (for k — n — 1) and [Ke, p. 283] (for any < k < n). 

For y = [y 1 ...y m ] E mt n ,m, let L(y) = lin(yi, ...,y m ) be the linear 
hull of the n-vectors y\ . . . y m , that is the smallest linear subspace con- 
taining y 1 . . . y m . Suppose that rank(w) = L Then dimL(y) = £ < m. 

Theorem 4.5. Let f E L 1 (9}t n)m ), n — k > m. If y E %K n ,m, and C 
an (n — k) -dimensional plane in IR n containing L(y), then for any 
orthonormal frame £ G V n ^ n -k spanning (, there exists b E Tl n -k,m so 
that y — £b. In this case 



(4.19) 



exp(itr(6't))/(e,t) dt, 



Mr, 



or 



(4.20) w)(e&) = ^[/(e, •)](&)> ^V/=> &ew n _ fc , m . 

Proof. Since each vector (j = 1, . . . , m) lies in (, it decomposes as 
%' = &j for some G R n_A: . Hence y = £5 where 6 = [&i . . . 6 m ] G 
attn-jfc.m- Thus it remains to prove (|OD|l . By (fODjl . 



nf(£, •)](&) = y exp(itr(6't))(ft y /(# 



it 



If x = g£ 



u 
t 



then, by (1431) . 



it 




u 


t 




t 



e 





In—k 



E V r 



n,n—ki 
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and the Fubini theorem yields 

F\f(& •)](&) = J exp(ztr(6'e'x)) /(*) dx = (Tf)(^b). 

□ 

Remark 4.6. It is clear that matrices £ and 6 in ()4.19|) are not uniquely 
defined. In the case rank(y) = m, one can choose £ and b as follows. 
By taking into account that n — k > m, we set 



Uq 







G vn-Jfc.mj ^0 





I in 



so that £o«o = ^o- Consider the polar decomposition 

y = wr 1/2 , u £ K,n-fc, r = y'y E V rn , 
and let ^ be a rotation with the property g v Vo = v. Then 
V = vr 1/2 = g v v r 1/2 = g v £ u r 1/2 = £6, 

where 



(4.21) £ = g v £ e K,„- fc , b = u^r 1 ' 2 e m n 



-k,m- 



Theorem 4.7. 

(i) If n — k > m, then the Radon transform f —*■ f is injective on the 
Schwartz space S(wt nim ) ! and f can be recovered by the formula 



n — m — 1 



fix) = - — - — / \r\ 2 dr 

J v / (2n) nm J 1 1 

Vm 

(4.22) 

x j exp(-ztr(xV 1 / 2 ))(^/(^£o,-))(^or 1/2 )^. 

(ii) For n — k < m, the Radon transform is n on- injective. 

Proof. By Theorem 14.51 given the Radon transform / of / £ S(Wl n>m ), 
the Fourier transform (J r f)(y) can be evaluated at each point y £ 
%Rn,m by the formula (j4.19|) . so that if / = then Tf = 0. Since 
T is injective, then / = 0, and we are done. Remark 14.61 allows to 
reconstruct / from /, because (I4.21j) expresses £ and b through y £ 
ffl n ,m explicitly. This gives (|4.22|h To prove (ii), we denote 

£n,m = {x £ VK, m ■ rank(x) = m}. 
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This set is open in £Dt„, m . Let ip be a Schwartz function with the Fourier 
transform supported in Z n>m . By (|4.2U|) . 

(4.23) ^(f, ■)](&) = to) = VZeV n , n „ k ,Vbewi n - kim , 

because Sfi ^ £ n>m (note that since n — k < m, then rank(£6) < m). By 
injectivity of the Fourier transform in ([4.23)1 . we obtain t) = V£, t. 
Thus, for n — k < m, the injectivity of the Radon transform fails. □ 

Remark 4.8. After the paper had been finished, we became aware of 
another account of the topic of this subsection in [Shi], [Sh2], written 
in a different manner. Nevertheless, Remark 14.61 formula (|4.22j) and 
the statement (ii) of Theorem 14. 71 seem to be new. 

Remark 4.9. For n — k > m, the dimension of the manifold G(n, k, m) 
of all matrix fc-planes in 97l nim is greater than that of the ambient 
space 9Jln tm , and the inversion problem is overdetermined. In the case 
n — k = m both dimensions coincide. The problem of reducing overde- 
terminicity by fixing a certain "invertible" mn-dimensional complex of 
matrix planes, was studied in [Sh2]. 



5. The dual Radon transform and the Fuglede formula 

Definition 5.1. Let r = r(£, t) be a matrix plane ([4.1J1 . (£, t) G 

V njri _k x %Rn-k,m- The dual Radon transform (p(x) assigns to a function 
(f(r) on G(n, k,m) its mean value over all matrix planes r through x. 
Namely, 




This means that 

(5.1) cp(x) = ^— [ <p(Z,Z'x)d£ 

Vn ,n — k 

SO(n) 

The mean value <p(x) apparently exists for all x G 3Jl nim if ip is a 
continuous function. Moreover, <p(x) is finite a.e. on OT nim for any 
locally integrable function ip [OR]. 

Remark 5.2. The dual Radon transform <p(x) of a function <p(t), t G 
G(n,k,m), is independent of the parameterization r = r(£, t) in the 



'■n—k 



GK 
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sense that for any other parameterization r = r(£#', 6t), 9 G 0(n — k), 
(see Sec. 14.1)1 . ()5.1j) gives the same result: 

— ^- [ ^e'M'x)di = ^— [ <p(s 11 &x)dt 1 = <p(x). 

&n,n—k J ^n s n~k J 

Mi,n— k ~Vn,n — k 

Lemma 5.3. The duality relation 

1 



(5.2) 



f{x)(p{x)dx 



0~n,n—k 



d£ / <p(U)f(U)dt 



Vn,n-k 97L 



holds provided that either side of is finite for f and replaced by 
\f\ and \<p\, respectively. 



Proof. By the right hand side of JOJ) is 

(5.3) -J— J di J <p{t,t)dt J f(g 6 U t ^j 



Or. 



du. 



9Jl fc 



u 

t 



we have 



Changing variables x — g% 

Hence, by the Fubini theorem, ()5.3j) reads 
1 



u 




u 


t 




t 



t. 



On,n—k 



d£, / <p(£,€'x)f(x)dx = / <p{x)f(x)dx. 



n . r, 



Now we state the main theorem. 



Ttn, 



□ 



Theorem 5.4. Let f e L p (97t njm ), 1 < p < (n + m - l)/(k + m - 1), 
or f is a continuous function satisfying f(x) = 0(\I m + x'x\~ x ^ 2 ), 
A > k + m — 1 . Then 

(5.4) c(/y (*) = (/*/)(*), c = 2- fcm vr^/ 2 r m (^)/r m (^ 
(the generalized Fuglede formula). 

Proof. The proof given below is applicable to any locally integrable 
function / for which either side of (|5.4|) is finite provided / is replaced 
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by |/|. Let f x (y) = f(x + y). By (j£5) and gH, 



0~n,n—k 



du 



0~n,n—k 



n,n — k 



f x + 7 



o 



g?7- 



50(n) 



This coincides with c 1 (I k f)(x) for k < m, see Definition 13.31 If > 
m, d = (m + l)/2, we pass to polar coordinates and get 



(/) V (*) 



T m J dv J \r\ k/2 - d dr J f x ^7 

V k . m Tm 50(n) 

2 " mCTfc - m J \r\ k ' 2 ~ d dr j f x {wr l ' 2 )dw 

/ \(k~n)/2 



yj.1/2 

o 



o?7, 



/(a: + y)|j/y| 



9K„ 



□ 



Corollary 5.5. (cf. Conjecture 3.4) For / e L p (m ritm ) and k G N, the 
Riesz potential (I f)(x) is finite a.e. on OT„ im if and only if 

(5.5) 1 <p < (n + m — l)/(k + m— 1). 

If f is a continuous function satisfying f(x) = 0(\ I m + x'x\~ x ^ 2 ), then 
(I k f)(x) is finite for all x G 97t n , m if an d on ^U 2/ A > k + m — 1. 

This statement holds by Theorem 14.41 

6. Inversion problem for the Radon transform 

The Fuglede formula c(/) v = I k f reduces the inversion problem for 
the Radon transform to Riesz potentials. This is exactly the same 
situation as in the rank-one case. Whereas for the ordinary fc-plane 
transform and Riesz potentials a variety of pointwise inversion formulas 
is available in a large scale of function spaces [Rul], [Ru2], in the higher 
rank case we cannot obtain pointwise inversion formulas rather than for 
Schwartz functions via the Fourier transform (see (4.22)) or in terms of 
divergent integrals understood somehow in the regularized sense. This 
is still an open problem. 



18 



E. OURNYCHEVA AND B. RUBIN 



Below we show how the unknown "rough" function / can be re- 
covered in the framework of the theory of distributions. First we 
specify the space of test functions. From the Fourier transform for- 
mula (h a , f) = (|y|~ a , (JF/) (?/)), it is evident that the Schwartz class 
S = «S(SDT rljm ) does not suit well enough because it is not invariant un- 
der multiplication by \y\^- To get around this difficulty, we follow an 
idea of V.I. Semyanistyi [Se] suggested for m = 1. Let ^ = ^/(DJl nim ) 
be the subspace of functions ip(y) € S vanishing on the set 

(6.1) {y : y E 97l„ im , rank(y) < m} = {y : ye Tt n>m , \y'y\ = 0} 

with all derivatives (the coincidence of both sets in ()6.1|) is clear because 
rank(y) = ra.nk(y'y), see, e.g., [FZ], p. 5). The set \1/ is a closed linear 
subspace of S. Therefore, it can be regarded as a linear topological 
space with the induced topology of S. Let $ = $(9Jt„ jm ) be the Fourier 
image of Since the Fourier transform T is an automorphism of S 
(i.e., a topological isomorphism of S onto itself), then $ is a closed 
linear subspace of S. Having been equipped with the induced topology 
of S, the space $ becomes a linear topological space isomorphic to ^ 
under the Fourier transform. We denote by $' = $'(OT„ )m ) the space 
of all linear continuous functionals (generalized functions) on $. Since 
for any complex a, multiplication by \y\^* is an automorphism of ^, 
then, according to the general theory [GSh], 1°, as a convolution with 
h a , is an automorphism of <3>, and we have 

^I a f](y) = \y\ m a m(y) 
for all ^'-distributions /. 

In the rank-one case, the spaces $, their duals and generaliza- 
tions were studied by P.I. Lizorkin, S.G. Samko and others in view of 
applications to the theory of function spaces and fractional calculus; 
see [Sa], [SKM], [Rul] and references therein. 

The Fuglede formula ()5.4|) , Theorem 14 A\ and Corollary 15.51 imply 
the following 

Theorem 6.1. Let f G L p (wi ntm ), 1 < p < (n + m - l)/{k + m - 1) 
or f is a continuous function satisfying f(x) = 0(\I m + x'x\~ x / 2 ) for 
some A > k + m — 1. Then the Radon transform g = f is well defined, 
and f can be recovered from g in the sense of -distributions by the 
formula 

(6.2) (/,0) = c(^,/- fc 0), 
where 

(i-VX*) = (^M^x*), c = 2- fcm vr- fc ™/ 2 r m (^)/r m (|). 
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Remark 6.2. For k even, the Riesz potential I k f can be inverted (in the 
sense of ^'-distributions) by repeated application of the Cayley-Laplace 
operator A m = det(<9'<9), d = (d/dxij) [Kh]. In the Fourier terms, 
this operator agrees with multiplication by (— l) m |?/|^, and therefore, 
(-l) m A m rf = I a ~ 2 f in the $'-sense. 
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